System and method for enhanced data analysis with video enabled software tools for medical environments

ABSTRACT

Medical software tools platform utilizes a surgical display to provide access to specific medical software tools, such as medically-oriented applications or tools, that can assist those in the operating room, such as a surgeon or surgical team, with a surgery. In particular, an optical sensor located within an endoscopic camera may register subtle differences in color characteristics reflected from a tissue surface and in turn transmit the information to a medical image processing system. Moreover, the new tools are intended to help surgeons better determine the boundaries between healthy and diseased regions during surgical procedures. Various tools are intended to be used in procedures where indocyanine green (ICG) fluorescent dye is used. Intraoperative fluorescence imaging is commonly used during minimally invasive procedures to enable surgeons to visualize tissue perfusion and anatomical structures. The term perfusion refers to the passage of blood and tissue fluid through the capillary bed. Intraoperative fluorescence imaging can also be used to improve visualization of vessels and structures, which, in turn, may reduce the risk of complications during minimally invasive surgeries.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.16/677,584, filed Nov. 7, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/958,944, filed Apr. 20, 2018, and issued as U.S.Pat. No. 10,507,065 on Dec. 17, 2019, which is a continuation in part ofU.S. patent application Ser. No. 15/789,948, filed Oct. 20, 2017, whichis a continuation in part of U.S. patent application Ser. No.15/652,031, filed Jul. 17, 2017, issued as U.S. Pat. No. 10,433,917, onOct. 8, 2019, which is a continuation in part of U.S. patent applicationSer. No. 15/456,458, filed Mar. 10, 2017, which is a continuation inpart of U.S. patent application Ser. No. 15/377,817, filed Dec. 13,2016, which is a continuation of U.S. patent application Ser. No.14/107,329, filed Dec. 16, 2013, and issued as U.S. Pat. No. 9,526,586on Dec. 27, 2016, which claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/865,037, filed Aug. 12, 2013. This applicationalso claims the benefit of and is a continuation in part of U.S. patentapplication Ser. No. 15/170,575, filed Jun. 1, 2016, issued as U.S. Pat.No. 10,142,641, on Nov. 27, 2018, which is a continuation of U.S. patentapplication Ser. No. 13/430,489, filed Mar. 26, 2012, which is acontinuation-in-part of U.S. patent application Ser. No. 12/776,048,filed May 7, 2010, and issued as U.S. Pat. No. 8,266,333, on Sep. 11,2012, which claims the benefit of U.S. Provisional Patent ApplicationSer. No. 61/182,624, filed May 29, 2009, and 61/234,577, filed Aug. 17,2009. Each of the foregoing applications is hereby incorporated byreference.

TECHNICAL FIELD

The technology disclosed herein relates to medical software tools and,in particular, some embodiments relate to systems and methods for asoftware tools platform in a medical environment, such as a surgicalenvironment, incorporating enhanced data analysis.

BACKGROUND OF THE INVENTION

Disclosed herein is an updated medical software tool platform, akasurgeon's dashboard, comprised of a new set of desktop tools (akawidgets) developed to help medical professionals identify subtle changesin tissue. Various tools are intended to be used in procedures whereindocyanine green (ICG) is used. ICG is a fluorescent dye and a markerin the assessment of the perfusion of tissues and organs. The termperfusion refers to the passage of blood and tissue fluid through thecapillary bed. Moreover, the new tools are intended to help surgeonsbetter determine the boundaries between healthy and diseased regionsduring surgical procedures. Intraoperative fluorescence imaging iscommonly used during multiple minimally invasive procedures to enablesurgeons to visualize tissue perfusion and anatomical structures. Thefluorescent imaging agent binds to protein in blood and is metabolizedand excreted by the liver thereby providing laparoscopic visualizationof the hepatic artery and bile ducts. These dyes enable a precise visualassessment of blood flow in vessels, as well as the quality of tissueperfusion in, for example, colorectal, esophageal and bariatricsurgeries. Intraoperative fluorescence imaging can also be used toimprove visualization of vessels and structures, which, in turn, mayreduce the risk of complications during minimally invasive surgeries.

Various desktop tools are designed to be used in conjunction with ICG toenhance a surgeon's ability to visualize and analyze tissue perfusionand structural anatomy intraoperatively and in real-time. In certainsituations, the tools can be used to help the surgeon visualize andobjectively analyze different levels of blood flow in tissue. Moreover,various desktop tools may assist surgeons in making critical decisionsin the operating room, which can potentially reduce rates ofpostoperative complications and decrease healthcare costs.

The new medical software tools include 1) ICG Visualization, 2) InstantReplay, 3) Height Mapping, 4) Grid Tool, 5) Perfusion Visualization andQuantification, 6) Color Collaboration. Each of the individual tools aredepicted in the figures below and described in the narrative thatfollows.

SUMMARY OF THE INVENTION

Various embodiments of the disclosed technology provide a medicalsoftware tools platform that utilizes a surgical display to provideaccess to medical software tools, such as medically-orientedapplications or widgets, that can assist those in the operating room,such as a surgeon and their surgical team, with a surgery. For variousembodiments, the medical software tools platform and its associatedmedical software tools are presented on a surgical display (e.g., beingutilized in an operating room) over an image stream provided by asurgical camera (e.g., in use in the operating room) or other medicaldevice that generates image streams. An image stream can include videoor a series of static images (e.g., medical ultrasound device). Variousmedical software tools can provide features and functions that canfacilitate integration of equipment in an operating room or add medicalcontext awareness to anatomic structures presented in the image streamfrom the surgical camera.

Medical video display panels and multiscreen displays are commonlyemployed in such contexts as hospital operating theaters or any facilitywhere surgical operations are carried out in a sterile environment. Theyare used to display visual information from data streams such assurgical imagery from an endoscope, patient vital signs, patient medicalrecords, clinical imaging data (patient CT scans, MRIs, etc.), outputsfrom other operating room equipment, and operating room environmentalstatus. Surgical displays and multiscreen video displays provide asurgeon and their surgical team with visual information that can beautomatically updated, or can be used to enable collaboration amongviewers. Where a surgical display or multiscreen video display is usedfor group collaboration, there is generally a requirement that the grouphas the ability to update and reconfigure the visual informationdisplayed, which is usually facilitated through a video switch.Traditional video switches are controlled through a switch box, akeyboard, or a local connection (via an RS-232 port or Ethernet port)and have only a single point for control access. In some contexts,visual data streams to a single large panel video display or amultiscreen display configuration are provided by two or more computersystems, each being controlled by a computer operator (i.e., user) usingsuch input/output (IO) devices as keyboards, mice, and as video monitor.

One of ordinary skill in the art would understand that, depending on theembodiment, either the image stream input interface, the image streamoutput interface, or both may utilize unidirectional communication orbidirectional communication with input devices and output devices. Forexample, a system may be configured to receive control information froma controller interface device via an image stream output interface, orto send control information to an image source via an image stream inputinterface. In another example, the control information is received bythe system through the Display Data Channel (DDC). Depending on theembodiment, the system may be configured to send control information toa device external to the system, through the image stream inputinterface.

In some embodiments, the switching matrix may selectively map an imagestream input interface or a processed image stream in real-time. Infurther embodiments, the switching matrix may selectively map the imagestream input interface to more than one image processing module or tomore than one image stream output interface simultaneously.Additionally, in some embodiments, the switching matrix may selectivelymap the processed image stream to more than one image processing moduleor to more than one image stream output interface simultaneously. Inother embodiments, the switching matrix may selectively map an imagestream input interface or the processed image stream based on acriterion. For example, the switching matrix may selectively map animage stream input interface or a processed image stream based on itssource or content. In another example, the switching matrix mayselectively map a processed image stream based on the results of apreceding image processing module. Depending on the embodiment, theimage processing module may have the capability of processing aplurality of image streams in parallel. The image stream interface forsome embodiments may be configured to receive the image stream from animage stream capture device, an image stream playback device, a computersystem, a sensor device or a medical device (e.g., endoscope). The imagestream output interface for some embodiments may be configured to outputto a display (e.g., liquid crystal display monitor), a computer system,or recording device (e.g., digital video recorder). Further, in someembodiments, the system may be configured to output an image streamthrough a virtual display.

In numerous embodiments, the system further comprises a data inputinterface, wherein the switching matrix is further in communication withthe data input interface such that the switching matrix can furtherselectively map the data input interface to the image stream outputinterface or to the first image processing module. For some suchembodiments, the image stream input interface may comprise the datainput interface.

For some embodiments, dynamic selection with respect to inputs meansthat one or more inputs can be selected or unselected in real time forrouting to one or more image processing elements. For additionalembodiments, dynamic and iterative selection with respect to processingelements means that a selected image stream can be routed to one or moreimage processing elements simultaneously and in real time. The routingmay be based upon criteria relating to the image stream's source, theimage stream's content, or on the processing results of a precedingimage processing element. The output of an image processing element maybe directed back to the system or method for routing to a subsequentimage processing element, or to one or more image outputs that supply animage stream to an output device (e.g., display, image recorder, orother image processor, or transmission device).

An exemplary system for switching control between a plurality ofcomputer systems, comprising a plurality of image stream inputinterfaces, wherein a first image stream input interface of theplurality of image stream input interfaces is configured to couple witha first computer system of the plurality of computer systems, andwherein a second image stream input interface of the plurality of imagestream input interfaces is configured to couple with a second computersystem of the plurality of computer systems. The system may furthercomprise a plurality of computer input device interfaces (e.g.,Universal Serial Bus [USB], PS/2, AT connector, Bluetooth, Infrared[IF], or FireWire), wherein a first computer input device interface ofthe general plurality of computer input device interfaces is configuredto couple with the first computer system, and wherein a second computerinput device interface of the plurality of general computer input deviceinterfaces is configured to couple with the second computer system.

The system may additionally comprise: an ICG visualization tool thatenables the visualization of perfusion of the ICG agent to help identifyhealthy tissue; an Instant Replay tool that enables a surgeon to replaya portion of the surgical video and to slow down the visual presentationof the absorption process to better characterize the differences inneighboring tissue; a Height Mapping tool that allows mapping of greenintensity to reveal greater detail in the presence of ICG; a Grid Toolthat overlays a dynamically adjustable grid over a selected area of thesurgical video. The number of grid lines along the x and y axes can beincreased or decreased by user on the fly; a Perfusion Visualization andQuantification tool that brings up a panel that quantifies the degreeand rate of selected tissue to absorb and dissipate contrast fluid (ICG)injected into a patient's blood stream; and a Color Collaboration toolthat allows the surgeon to personalize gamma adjustments in real time.The colors are adjusted and displayed to a standard unique to eachindividual user.

Other embodiments provide for a computer readable storage medium havinginstructions embedded thereon to cause a processor to perform operationssimilar to those described above with respect to the various systems andmethods in accordance with the present invention. Other features andaspects of the disclosed technology will become apparent from thefollowing detailed description, taken in conjunction with theaccompanying drawings, which illustrate, by way of example, the featuresin accordance with embodiments of the disclosed technology. The summaryis not intended to limit the scope of any inventions described herein,which are defined solely by the claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the overallprocessing system that may be used in implementing various features ofembodiments of the disclosed technology.

FIG. 2 is a block diagram illustrating an example of the imageprocessing system that may be used in implementing various features ofembodiments of the disclosed technology.

FIG. 3 is a block diagram illustrating an example medical software toolsplatform in accordance with some embodiments of the technology describedherein.

FIG. 4 . ICG Visualization, a tool that enables the visualization ofperfusion of the ICG agent to help identify healthy tissue.

FIG. 5 . Instant Replay, a tool that enables a surgeon to replay aportion of the surgical video and to slow down the visual presentationof the absorption process to better characterize the differences inneighboring tissue.

FIG. 6 . Height Mapping, a tool that allows mapping of green intensityto reveal greater detail in the presence of ICG.

FIG. 7 . Grid Tool, a tool that overlays a dynamically adjustable gridover a selected area of the surgical video. The number of grid linesalong the x and y axes can be increased or decreased by user on the fly.

FIG. 8 . Perfusion Visualization and Quantification, a tool that bringsup a panel that quantifies the degree and rate of selected tissue toabsorb and dissipate contrast fluid (ICG) injected into a patient'sblood stream.

FIG. 9 . Color Collaboration, a tool that allow the surgeon topersonalized gamma adjustments in real time. The colors are adjusted anddisplayed to a standard unique to each individual user.

FIG. 9.1 is an example of different types of color processing for theColor Collaboration tool.

FIG. 10 . is a block diagram illustrating the ICG Visualization modulewithin the medical software tools platform.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a block diagram illustrating an example of the overallprocessing system that may be used in implementing various features ofembodiments of the disclosed technology. In accordance with thepreferred embodiment of the present invention, the processing system 100consists of processor elements such as: a central processing unit (CPU)102; a graphics processing unit (GPU) 104; and a field programmable gatearray (FPGA) 106. The processing system 100 may be used to retrieve andprocess raw data derived from a surgical camera 110 or a data storagedevice, such as a medical archive 108. The surgical camera 110 ormedical archive 108 transmits a data stream to the processing system100, whereby that data is processed by the CPU 102. The FPGA 106,connected to the CPU 102 and the GPU 104, simultaneously processes thereceived data by using a series of programmed system algorithms 118,thus functioning as an image clarifier within the processing system 100.The GPU 104 communicates with the user interface 112 to display thereceived data from the medical archive 108. The GPU 104 enables the userinterface to then communicate the data to connected input devices 114and output devices 116. The user interface 112 can communicate tomultiple input 114 and output devices 116 simultaneously. An inputdevice 114 can include, for example, a keyboard, touchscreen or voiceactivated device. An output device 116 can include, for example, a videodisplay, a digital video recorder (DVR) or universal serial bus (USB).

FIG. 2 is a block diagram illustrating an example of the imageprocessing system that may be used in implementing various features ofembodiments of the disclosed technology. In accordance with thepreferred embodiment of the present invention, the image processingsystem 200 consists of three components that process image data receivedfrom a sensor 202 in order to send that data to a display or videorouter 210. The three components of the image processing system 200 are:camera head video pre-processing 204; real time video enhancement 206;and the video display transport 208 function. Image data is collected bya sensor imaging device 202, and is then transmitted to the camera headvideo pre-processing component 204 within the image processing system200. This data may be, for example, a raw video image that ispre-processed using various image processing algorithms. Imagepre-processing may also include software modules for image registrationand segmentation to optimize the video data and communicate via thesystem bus 212 with the internal system processors 214: the CPU; GPU;and FPGA.

The pre-processed image data is transmitted to the real time videoenhancement 206 component, whereby the image data is enhanced to improveclarity or highlight certain details. Once the image data resolution hasbeen enhanced, the video display transport 208 component completes imagepost-processing, formatting from the initial sensor resolution to theeventual display resolution, for example, enhancing the video data to1080p HD or 4K display resolution or using software modules such asvideo cross conversion, scaling and adding graphic overlays. Theprocessed image data is then transmitted from the image processingsystem 200 to the display or video router 210. The video displaytransport also saves the processed image data to the processing systemmemory 216 that can consist of internal and external memory storage.

FIG. 3 is a block diagram illustrating an example medical software toolsplatform system in accordance with some embodiments of the technologydescribed herein. The medical software tools platform provides access tomedically-oriented applications (apps) or tools that can assist membersof the surgical team during an operation. For example, many surgeriesutilize ICG to enable surgeons to visualize tissue perfusion andanatomical structures, however, the camera sensor captures much morecolor detail that the human eye can discern. Therefore, the use of anICG visualization tool can provide a more accurate and objective measureof the rate of perfusion. An instant replay tool enables the users toreplay a portion of the surgical video in slow motion in order to moreclosely observe the process of perfusion. A height mapping tool providesa simulated 3D visualization that enables the user to quickly andaccurately identify areas of highest absorption of the ICG fluorescentdye. A grid tool brings up an overlay that draws grid lines to providean x,y location reference. This enables a surgeon to precisely mark thelocation of an anomaly or area of interest for future reference. Aperfusion visualization and quantification tool precisely measure therates of rise and decay of the of fluoresced pixel intensity. This isintended to assist a surgeon in identifying the borders between healthyand diseased tissue. A color calibration tool allows users to configurea color preference so that colors are adjusted and displayed to astandard unique to each individual. This enables color blind surgeons tobetter visualize information on the surgical display and perceive subtlechanges in color.

In accordance with the preferred embodiment of the present invention,the medical software tools platform system 300 includes: an image streaminterface module 302; a user interface overlay module 304; medicalsoftware tools 310; a medical device interface module 306; and an imagestream processing system interface module 308. The medical softwaretools platform system 300 may be integrated, in whole or in part, into avideo display or an image stream processing system utilized in anoperating room. The image stream interface module 302 may receive animage stream acquired by a surgical camera or the like. Depending on theembodiment, the image stream may be received directly from the surgicalcamera, or may be provided by way of one or more components, such as animage stream processing system. The image stream received from the imagestream interface module 302 may vary in resolution, frame rate, format,and protocol according to the surgical camera or the image streamprocessing system providing the image stream.

The user interface overlay module 304 may provide a user interface tothe medical software tools platform system 300, which may include one ormore graphical user interface (GUI) elements presented over the imagestream received through the image stream interface module 302. For someembodiments, the user interface comprises a bottom toolbar configured tobe presented over the image stream, and configured to provide access tovarious medical software tools 310 available through the medicalsoftware tools platform system 300.

The medical software tools platform system 300 may include one or moremedical software tools, such as medically-oriented applications orwidgets, which can be utilized with respect to the image stream beingreceived through the image stream interface module 302. The medicalsoftware tools 310 include but are not limited to: a ICG Visualizationmodule 312; an instant replay module 314; a height mapping module 316; agrid tool module 318; an perfusion visualization and quantificationmodule 320; a color calibration module 322.

The medical device interface module 306 may facilitate communicationbetween the medical software tools platform system 300, one or more ofthe medical software tools 310, and one or more various medical devicesutilized in an operating room. The image stream processing systeminterface module 308 may facilitate communication between the medicalsoftware tools platform system 300 and an image stream processing systemutilized to process an image stream acquired by a surgical camera or thelike. Through the communication, the image stream processing systeminterface module 308 may transmit control data to an image streamprocessing system, or receive an image stream from a surgical camera asprocessed by the image stream processing system. The image streamprocessing system interface module 308 may include various datainterfaces, including wired or wireless network interfaces and serialcommunication interfaces.

ICG Visualization (FIG. 4 ) tool enables the visualization of perfusionof the ICG agent to help identify healthy tissue. The ICG Visualizationtool can be used in conjunction with other tools to enable the followingcapabilities: 1. Slow Motion—replaying the video in slow motion 2.Intensity—quantifying the degree of ICG absorption for defined areas ofinterest 3. Speed—measuring the time to full absorption for definedareas of interest 4. ICG Map—simulated 3D views of ICG absorption usingintensity or speed of ICG data.

In a preferred embodiment the ICG Visualization tool can be used inconjunction with the Slow Motion Replay tool 402 and the Height Mappingtool 404, which can be integrated into the ICG Visualization userinterface. In this implementation, a surgeon might use the ICGVisualization in conjunction with the Slow Motion Replay tool and theHeight Mapping tools in a four step process as follows.

In a first step, a user, such as the surgeon or an assistant, uses themouse to sweep an area of interest. The selected image segment is scaledand centered in the yellow working box as show in the upper left corner408.

As ICG is administered, the computer measures time to reach fullluminance. The example in FIG. 4 shows this time to be 12 seconds 410.

Video frames are cached in memory a full frame rate. Slow Motion replaysare available at selected speeds and use motion-compensated frameinterpolation for a smooth playback. An elevator bar on the right sideis used to control image zoom.

In a second step, an operator uses the mouse to create a box around oneor more regions of interest, as shown in FIG. 4 on the right where threeyellow boxes are selected 406.

The intensity and the time to reach full intensity for each region isgraphed using the ICG channel to provide a quantitative comparison 410.

Selected regions are described by outline boxes (dashed yellow areas inthe example 406). Unchecking the form box labeled, “Show Regions” hidesthe outlines 414. Region outlines can be moved and resized with thegraphs updated in real time.

In a third step, the system quantifies perfusion based on intensity(absorption) and speed 404.

In a fourth step, the user has the option to invoke slow motion instantreplays at selectable speed so that the details of the process ofperfusion can be reviewed and studied 402.

The user also has the option to invoke Height Map tool which shows ICGdata in simulated 3D to demonstrate areas of highest perfusion. Bothintensity and speed can be mapped and the image can be rotated 404.

Instant Replay (FIG. 5 ) tool enables a surgeon to replay a portion ofthe surgical video. An example above described how Instant Replay can beused to review the speed of ICG contrast agent perfusion to provide thesurgeon an opportunity to slow down the visual presentation of theabsorption process to better characterize the differences in neighboringtissue. FIG. 5 shows how Instant Replay works. Video from the endoscopeis recorded and stored in a special purpose computer that implementsvarious image processing algorithms 500. The last n seconds of videoframes are stored on a circular video frame buffer within the computer502. Instant Replay uses video frames from the computer circular memorybuffer, playing frames at selected frame rates. The user sets the recalltime period and speed from slow motion to speed forward 504. The usercan select algorithms to apply to the video as it is being displayed506. The user can also select levels of contrast and brightness for thereplay 508.

The Height Map tool is shown in FIG. 6 . To use the tool, the user firstsweeps an area of interest with the mouse. The selected area is enlargedand rendered as a “Picture-In-Picture” insert 600. The insert simulatesa 3D view by mapping the area's pixels onto an elevation grid using thepixel intensities to create a height map. The user can experiment withdifferent views and degrees of detail by rotating the insert in 3D usinga mouse, adjusting for more or less detail with the window/level sliders602, and by scaling with the vertical bar 604. The Pulse button 606animates the pixel heights in relation to the intensity change resultingfrom the pulse of pumping blood into the local vascular network. Thiscreates a visual effect of pixels dancing in synchronization with thepatient's heartbeat. The rise and fall height and rate of change inpixel intensity demonstrate the different responses of diseased andhealthy tissue.

The Grid Tool is shown in FIG. 7 . Selecting the Grid tool brings up anoverlay that draws grid lines to provide an x,y location reference 710.The icon includes a “minus” and a “plus” touch point to grow or shrinkthe grid resolution 700. Double clicking brings up a sub-menu 702 toselect full or border grid lines and select line color and opacity.Default settings from the Personal Preferences are used when the tool isinitialized.

The grid is dynamically adjustable and the number of grid lines in x andy can be increased or decreased by user on the fly. The Grid Lines areselectable such that the user can select lines at borders 704, acrossfull image 708, or none. Users can configure the attributes of the gridlines to set line weight, color and opacity 702. In multi-displayenvironments, the user can select which displays have grid lines. Theuser can configure the systems to display the coordinates of the visiblecursor location 712. This visual confirmation of cursor location caneliminate doubt and ease cognitive burden for the user. The system alsooffers the ability to save the personal settings for each user,including an option to recall “Last Used” settings.

The Perfusion Visualization and Quantification tool is shown in FIG. 8 .Selecting the Perfusion Visualization and Quantification tool brings upa panel that quantifies the degree and rate of selected tissue to absorband dissipate contrast fluid (ICG) injected into a patient's bloodstream 800. The tool displays both endoscopic video channels (RGB 802and IR 804) and a third fused image with the IR illuminated tissuetransformed to fluorescent green for overlay on the RGB image 806. Thepurpose is to assist the surgeon in characterizing the state of thetissue. Default settings from the Personal Preferences are used when thetool is initialized.

The user sweeps an area of interest 806 to see a graph of the averagepixel intensity (Y-axis) plotted against time (X-axis) 800. The curveprovides a quantitative measure of the tissue's absorption and decayrate for comparison with other areas to locate the healthiest tissue.Graphs respond in real time and may be moved about the image to exploreresponses. Graph results are plotted in different colors for comparisonpurposes 800.

The graph 800 provides an objective method to quantify the perfusionprocess using the degree of absorption and also the time to reachmaximum intensity and the time to dissipate the agent. Without thistool, surgeons must rely on subjective measurements of color intensityand time.

The tool enables a user to measuring rise time, the time it takes for afluoresced pixel intensity to achieve maximum luminance, and the decaytime, the time it take for a fluoresced pixel intensity to achieveinitial state. The instant replay feature allows the user to view thefluorescing period at selective speeds to better observe the process. Agraphical representation shows the amplitude and time of the fluorescingevent in real time 800. The tool is able to capture and compare multiplesamples using software mechanisms to collect and compare multiplesamples 806, 800.

The Color Collaboration tool, shown in FIG. 9 , presents the surgeonwith a choice of views of the tissue surface with a choice of color maps900. The goal is to more clearly characterize the margins of diseasedtissue, based on the individual color preference of a user.

FIG. 9.1 is an example of different types of color processing for theColor Collaboration tool. In accordance with the preferred embodiment ofthe present invention, surgeons use color to identify and characterizetissue in many ways. However individual humans perceive colordifferently and surgery frequently involves a team of specialists, eachperceiving color in their own way. Some form of color blindness effects˜8% of men and 0.5% of women and there are many types of colorblindness—FIG. 9.1 illustrates three common forms of color blindness.

By first sampling a surgeon's color perception against reference images900, the colors in the endoscopic video can be adjusted to present animage closer to a standard reference. Team members can have uniqueindividual adjustments effected simultaneously by using dedicateddisplays, or special eyeglasses equipped with miniature color displays,or a primary display that can switch between color tables on command.

Main Menu settings personalize the default choices for each surgeon. Thedefaults include the presentation format and color translation tables todisplay when the Color Map icon is selected 902. Mouse clicking aselection brings it to the primary display window where it is subject toother tools for further analysis as shown in FIG. 9 .

FIG. 10 is a block diagram illustrating the ICG Visualization module1012 within the medical software tools platform. In accordance with thepreferred embodiment of the present invention, an optical sensor 1002located within an endoscopic camera lens 1000 registers the colorcharacteristics reflected from a tissue surface. This image data 1004 istransmitted to the medical image processing system 1006, which can alsoreceive data from a device such as a patient heart rate monitor 1008.The medical image processing system 1006 processes and transmits theimage data 1004 from the optical sensor with data from the heart ratemonitoring device 1008 to the user interface 1010, whereby all sets ofdata are received through the optical signature measurement module 1012displaying the rate of change 1014 in the color characteristicssimultaneously with the patient vitals data 1016. This data is analyzed1018 in the module and the results of any anomaly detection 1020 aredisplayed to the user. The slight changes in one of more of the colorcomponents, and the speed at which they change, in relation to a sourcestimulus (heartbeat, breath, external stimulus) indicates the arrival ofblood, contrast agents, or oxygen absorption. Combinations of thecomponents can be used to characterize different types and states ofdisease and to identify the margins of tumors and diseased areas.

Although the disclosed technology is described above in terms of variousexemplary embodiments and implementations, it should be understood thatthe various features, aspects and functionality described in one or moreof the individual embodiments are not limited in their applicability tothe particular embodiment with which they are described, but instead maybe applied, alone or in various combinations, to one or more of theother embodiments of the disclosed technology, whether or not suchembodiments are described and whether or not such features are presentedas being a part of a described embodiment. Thus, the breadth and scopeof the technology disclosed herein should not be limited by any of theabove-described exemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, may be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives may be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

Embodiments presented are particular ways to realize the invention andare not inclusive of all ways possible. Therefore, there may existembodiments that do not deviate from the spirit and scope of thisdisclosure as set forth by appended claims, but do not appear here asspecific examples. It will be appreciated that a great plurality ofalternative versions are possible.

What is claimed is:
 1. A system for medical software tools, comprising:an image stream interface module configured to receive an image streamfrom a surgical camera via an image stream processing system over wiredor wireless network connection wherein said module includes a CPU, GPUand a FPGA; a user interface overlay module configured to provide a userinterface overlay adapted for presentation over the said image streamvia an image stream processing system and a graphical user interfacepresented over said image stream received through said image streaminterface module; an optical sensor located corresponding with saidsurgical camera for registering momentary changes in spectralcharacteristics reflected from a tissue surface under inspection; amedical software tools module configured to provide a medical softwaretool that utilizes a surgical display through said user interface,including medically-oriented applications or widgets that assistoperating room personnel including a surgeon and said surgeon's medicalteam, said medical software tool being configured to perform anoperation with respect to said image stream via software functions thatfacilitate the integration of operating room equipment with said medicalsoftware tools and provide an output adapted to be presented over saidimage stream; a medical image processing system for processing patientmedical data and corresponding said patient medical data with saidmomentary changes in spectral characteristics for generating opticalsignature data indicative of various patient conditions, wherein saidmedical software tool measures: a. changes in color intensity and b.rates at which said color intensities change in response to: heartbeatpushed blood, breathing pushed oxygen or light intensity or modulationfrom a light source; wherein a contrast dye is introduced associatedwith said tissue under inspection in the assessment of the perfusion oftissues; and, wherein an image stream enhancement function with either ade-haze function, a de-blur function, a shadow function, or a thermalturbulence function for enabling system users to obtain an optimizedview of a surgical procedure.
 2. A system according to claim 1 whereinsaid contrast dye is fluorescent dye.
 3. A system according to claim 1wherein said spectral characteristics associated with said patientmedical data may be displayed and then subsequently displayed over andover as desired by an operator of said system so that said operator canevaluate said patient medical data without regard for patient location.4. A system according to claim 1 wherein said patient data is shared viaa cloud-based data system.
 5. A system according to claim 1 wherein gridlines are output along with images associated with patient medical dataand wherein said grid lines are adjustable to be useful to a user of thesystem, and wherein the number of grid lines may be established alongany axis as selected by said user.
 6. A system according to claim 5wherein said grid lines correspond with prior surgical outcomes andwherein surgeons utilize said grid lines for enhanced surgical outcomes.7. A system according to claim 5 wherein said grid lines correspond to avalue corresponding to a cloud-based artificial intelligence databasefor enhancing surgical results.
 8. A system according to claim 5 whereinsaid grid lines correspond to a degree of opacity.
 9. A system accordingto claim 1 wherein a user selects a perfusion visualization andquantification tool to generate a visual panel for quantifying a degreeand a rate for selected patient tissue to absorb and dissipate saidcontrast fluorescent fluid injected into a patient's blood stream foroptimizing said images associated with patient medical data.
 10. Asystem according to claim 1 wherein a user selects a color collaborationtool for providing said user a choice of views of patient tissuesurfaces each corresponding with a color map choice.
 11. A systemaccording to claim 10 wherein said choice is based on user preferencewherein said user selects a color based on margins of diseased tissue.12. A method for providing medical software tools to surgeons,comprising: an image stream interface module configured to receive animage stream from a surgical camera via an image stream processingsystem over wired or wireless network connection, wherein said moduleincludes a CPU, GPU and a FPGA; a user interface overlay moduleconfigured to provide a user interface overlay adapted for presentationover said image stream via an image stream processing system and agraphical user interface presented over said image stream receivedthrough said image stream interface module; an optical sensor locatedcorresponding with said surgical camera for registering momentarychanges in spectral characteristics reflected from a tissue surfaceunder inspection; a medical software tools module configured to providea medical software tool that utilizes a surgical display through saiduser interface, including medically-oriented applications or widgetsthat assist operating room personnel including a surgeon and saidsurgeon's medical team, said medical software tool being configured toperform an operation with respect to said image stream via softwarefunctions that facilitate the integration of operating room equipmentwith said medical software tools and provide an output adapted to bepresented over said image stream; a medical image processing system forprocessing patient medical data and corresponding said patient medicaldata with said momentary changes in spectral characteristics forgenerating optical signature data indicative of various patientconditions, wherein the medical software tool measures: a. changes incolor intensity and b. rates at which said color intensities change inresponse to: heartbeat pushed blood, breathing pushed oxygen or lightintensity or modulation from a light source; wherein a contrast dye isintroduced associated with said tissue under inspection in theassessment of the perfusion of tissues; and, wherein an image streamenhancement function with either a de-haze function, a de-blur function,a shadow function, or a thermal turbulence function for enabling systemusers to obtain an optimized view of a surgical procedure.
 13. A methodaccording to claim 12 wherein said contrast dye is fluorescent dye. 14.A method according to claim 12 wherein said spectral characteristicsassociated with said patient medical data may be displayed and thensubsequently displayed over and over as desired by an operator of saidsystem so that said operator can evaluate said patient medical datawithout regard for patient location.
 15. A method according to claim 12wherein said patient data is shared via a cloud-based data system.
 16. Amethod according to claim 12 wherein grid lines are output along withimages associated with patient medical data and wherein said grid linesare adjustable to be useful to a user of the system, and wherein thenumber of grid lines may be established along any axis as selected bysaid user.
 17. A method according to claim 16 wherein said grid linescorrespond with prior surgical outcomes and wherein surgeons utilizesaid grid lines for enhanced surgical outcomes.
 18. A method accordingto claim 16 wherein said grid lines correspond to a value correspondingto a cloud-based artificial intelligence database for enhancing surgicalresults.
 19. A method according to claim 16 wherein said grid linescorrespond to a degree of opacity.
 20. A method according to claim 12wherein a user selects a perfusion visualization and quantification toolto generate a visual panel for quantifying a degree and a rate forselected patient tissue to absorb and dissipate said contrastfluorescent fluid injected into a patient's blood stream for optimizingsaid images associated with patient medical data.
 21. A method accordingto claim 12 wherein a user selects a color collaboration tool forproviding said user a choice of views of patient tissue surfaces eachcorresponding with a color map choice.
 22. A method according to claim21 wherein said choice is based on user preference wherein said userselects a color based on margins of diseased tissue.